Just adding a comment that fisheye projection could be along the lines of what Google has described in their spatial media specs: https://github.com/google/spatial-media/blob/master/docs/vr180.md

I haven't seen any specific recommendations from Apple, but for comparison, the "Highlining" video appears to be delivering 4320x4320, HRD10, 90fps content in fisheye projection at ~50Mbps for its best presentation.

By the way, it looks like there's a small bug in the side-by-side example code. In Coverter.swift around line 130, the else block that marks frameInput as finished should be in response to a nil sampleBuffer. Otherwise, the conversion will never end. To fix it, just move that else block to be a condition of the "if let sampleBuffer" logic (essentially un-indenting the logic one level...while fixing-up the parenthesis, of course).

Also, I ran across this post about creating a simple immersive 360º media player: https://medium.com/@pat.schnitzerb/how-to-make-a-360-videoplayer-for-visionpro-74e42e239e5 Note that the same technique can be used to project 180º media if you provide hemispherical geometry.

I'm not sure that I follow your question exactly, but there's no indication that I've seen that the framework natively supports playing back over/under-style content. You probably want to encode each eye to its own layer in a MV-HEVC file. Then, as far as I know, you need to build a player that takes the stereo MV-HEVC content and projects it onto a sphere/hemisphere in a RealityKit scene to see it in 3D.

As far as I know, there is no built-in method to playback half equirectangular (180º) or full equirectangular (360º) content. Many have looked (including me). Perhaps this is an upcoming feature of AVPlayer. In the meantime, it looks like you have to do what all other immersive players have done: create a RealityKit scene, add a hemisphere (180º) or full sphere (360º), and setup the AVPlayer to project to that surface. I don't know if this is the final/recommended way to do it, but it works well.